Surgical instrument path computation and display for endoluminal surgery

ABSTRACT

An endoscopic surgical navigation system comprises a path correlation module that can compute the path taken by an endoscope scope or other medical instrument during an endoscopic medical procedure and various related attributes and parameters, and can compute and display a correlation between two paths.

FIELD OF THE INVENTION

At least one embodiment of the present invention pertains to medicaldevices, and more particularly, to a technique for computing anddisplaying paths and associated attributes associated with a surgicalinstrument during endoscopic surgery.

BACKGROUND

To reduce the trauma to patients caused by invasive surgery, minimallyinvasive surgical techniques have been developed for performing surgicalprocedures within the body through very small incisions. Endoscopy is atechnique that is commonly employed in minimally invasive surgery.Endoscopy allows internal features of the body of a patient to be viewedthrough an endoscope, either directly or through video generated by avideo camera coupled to the endoscope. The endoscope typically can alsobe used as a conduit through which other surgical instruments can beinserted into the body.

Endoscopes can be of the rigid type or the flexible type. A rigidendoscope is typically inserted into the body through a small externalincision, as in laparoscopy, arthroscopy, etc. Flexible endoscopes, onthe other hand, are commonly used in procedures where the endoscope isinserted through a natural body orifice, such as the mouth or anus, asin gastroscopy or colonoscopy, respectively.

Endoluminal surgery is a newer form of minimally-invasive surgery, inwhich the surgical instrument (i.e., the endoscope or an instrumentinserted through it) initially enters the body through a natural bodilyorifice, such as the mouth. Typically a flexible endoscope is used. Theinstrument is then “threaded” through a natural body lumen, such as theesophagus, until its distal tip is close to the target anatomy. Oftenthe target anatomy is not in the immediate proximity of the orifice ofentry, however. Therefore, the surgeon must navigate the endoscope tothe target anatomy and may have to operate on portions of the anatomythat are not directly visible or are not easily visible.

Because endoscopes have limited field of view, localization of targetlesions and navigation to the desired areas through small entry pointscan be difficult. Furthermore, some parts of the body contain extremelysmall and/or complex structures that are difficult for a surgeon to seethrough an endoscope or in endoscopic video. The challenges becomelarger as the distance from the entry point to the target anatomyincreases, as is the case in endoluminal surgery.

BRIEF DESCRIPTION OF THE DRAWINGS

One or more embodiments of the present invention are illustrated by wayof example and not limitation in the figures of the accompanyingdrawings, in which like references indicate similar elements and inwhich:

FIG. 1 is a high level diagram of a system for performing endoluminalsurgery;

FIG. 2 schematically illustrates an example of a display of multiplecoregistered, multi-modal images;

FIG. 3 schematically illustrates an endoscopic surgery visualizationsystem that includes a Visual Navigation System (VNS);

FIG. 4 is a block diagram showing the elements of the VNS, according tocertain embodiments of the invention;

FIG. 5 illustrates an example of the overall process that can beperformed by the VNS while the VNS is in a particular operating mode tocoregister multi-modal images;

FIG. 6 is a block diagram showing the VNS in greater detail, accordingto certain embodiments of the invention;

FIG. 7 is a block diagram showing an example of the implementation ofthe multi-modal image coregistration module;

FIG. 8 is a block diagram showing an example of the user input portionof the user interface subsystem;

FIG. 9 is a block diagram showing the model generator according tocertain embodiments of the invention;

FIG. 10 schematically shows an example of a system configuration fortracking the position and orientation of the endoscope usingelectromagnetic sensors;

FIG. 11 schematically shows an example of a system configuration fortracking the position and orientation of the endoscope using opticalcurvature sensors;

FIG. 12 shows the construction of an optical curvature sensor that canbe used to track the endoscope;

FIG. 13 shows the use of the light channel in the endoscope to provide alight source for optical curvature sensors on the endoscope;

FIG. 14 shows the use of different length optical curvature sensors onthe endoscope;

FIG. 15 illustrates an example of a process for determining the currentposition and orientation of the distal tip of the scope;

FIG. 16 shows the tagging of video frames with position and orientationinformation;

FIG. 17 illustrates the relationship between the endoscope tip, theimage plane and the object being viewed;

FIG. 18 illustrates the placement of various video frames into a common3D coordinate space;

FIG. 19A shows a process of acquiring and processing video data toenable subsequent visual navigation;

FIG. 19B shows a process by which a user can visually navigate throughvideo that has been processed as in FIG. 19A;

FIG. 20 illustrates an endoscopic camera view of an anatomical object;

FIG. 21 shows a process of automatically measuring one or moreparameters of an anatomical feature;

FIG. 22 shows an example of a display of slice images, which can begenerated automatically by the VNS in response to movement of theendoscope;

FIG. 23 shows an example of a 3D rendering of a computed path taken byan endoscope during endoscopic surgery;

FIG. 24 shows an example of how the spatial relationship between twopaths can be displayed;

FIG. 25 illustrates an example of projecting a path onto threeorthogonal planes;

FIG. 26 illustrates the connection of endpoints of two paths;

FIG. 27 illustrates the computation of correlation between the currentposition of the endoscope tip and a reference path; and

FIG. 28 shows an example of a process for determining a correlationbetween two paths.

DETAILED DESCRIPTION

A visual navigation system for use in endoscopic surgery, particularly(though not exclusively) in endoluminal surgery, is described.References in this specification to “an embodiment”, “one embodiment”,or the like, mean that the particular feature, structure orcharacteristic being described is included in at least one embodiment ofthe present invention. Occurrences of such phrases in this specificationdo not necessarily all refer to the same embodiment.

In view of the challenges mentioned above, it is desirable to provide avisual navigation system to provide coupled three-dimensional (3D)visualization and navigation assistance to the surgeon in navigating anendoscope to the target anatomy, particularly during endoluminalsurgery. As described in greater detail below, therefore, according tocertain embodiments of the invention, a visual navigation system (VNS30) comprises a data acquisition subsystem, an endoscope trackingsubsystem, a registration subsystem, a data processing subsystem and auser interface subsystem. The data acquisition subsystem inputsintra-operative scan data from a medical scanning device during anendoscopic procedure. The tracking subsystem captures data representingpositions and orientations of a flexible endoscope during the endoscopicprocedure. The registration subsystem determines transformationparameters for coregistering the intra-operative scan data and the dataindicative of positions and orientations of the endoscope. The dataprocessing subsystem coregisters the intra-operative scan data and thedata indicative of positions and orientations of the endoscope based onthe transformation parameters and generates real-time image datarepresenting 3D internal views of a body that are coregistered with livevideo from an endoscopic video camera. The user interface subsystemreceives input from a user for controlling the system and providesoutput to the user.

The following definitions and explanations shall apply to terms usedherein:

“Coregistering” means bringing into a common coordinate space andorientation.

“Flexible” means designed to be flexed substantially without incurringdamage to the instrument (not just capable of being deformed).

A “flexible endoscope” is an endoscope, a substantial portion of thelength of which is flexible, including the distal end. (A “flexibleendoscope” can and usually does have a rigid proximal portion, or“base”.)

“Intra-operative scan data” is scan data acquired by a scan performedduring a particular endoscopic procedure on a body. This term does notinclude video acquired from an endoscopic video camera.

“Logic” can be or include (but is not limited to) any one or more of:special-purpose hardwired circuitry, programmable circuitry, software,firmware, or any combination thereof.

A “module” means any one or more of: special-purpose hardwiredcircuitry; software and/or firmware in combination with one or moreprogrammable processors; or any combination thereof.

“Positions” is synonymous with “locations”.

“Pre-operative scan data” is scan data acquired by a scan performedprior to a particular endoscopic procedure on a body.

During an endoscopic procedure on a body, the VNS inputs intra-operativescan data generated by a medical scanning device, such as an x-raycomputed tomography (CT) device, an MRI device, ultrasound imagingdevice, etc. The intra-operative scan data is representative of a regionof interest in the body. The VNS also captures data indicative ofpositions and orientations of a flexible endoscope during the endoscopicprocedure, from various sensors on the endoscope. The VNS furthergenerates real-time three-dimensional scan images of the region ofinterest based on the intra-operative scan data and/or the pre-operativescan data and the data indicative of positions and orientations of theflexible endoscope. The VNS coregisters the real-time three-dimensionalscan images with live video images generated by the endoscopic videocamera that is coupled to the endoscope. The VNS then causes thereal-time three-dimensional (volumetric) scan images and the live videoimages to be displayed coregistered on a display device.

The VNS can automatically detect movement of the flexible endoscopeduring an endoscopic procedure and, in response, identify a particularslice of scan data corresponding to a current location and orientationof the endoscope tip and cause an image of the slice to be displayed,and similarly cause other slices of scan data to be displayed inresponse to additional movements of the endoscope.

The VNS can also coregister and display the intra-operative scan datawith pre-operative scan data representative of the region of interest inthe body and generated prior to the endoscopic procedure by a medicalscanning device.

Another feature of the VNS is the ability to correct for barrel lensdistortion in the live video. Barrel lens distortion is divergence, inthe acquired endoscopic video, from the rectilinear projection ingeometric optics where image magnification decreases with increasingdistance from the optical axis.

Another feature of the VNS is a technique for employing model-fittingtechnique which enables a user easily to obtain in vivo measurements ofanatomical features in the body during an endoscopic procedure.

Yet another feature of the VNS is that it enables a user to visuallynavigate captured endoscopic video with six degrees of freedom. Thiscapability provides the user with control of a virtual camera (point ofview) that can be translated in three orthogonal axes in 3-D space aswell as allowing control of vertical panning (pitch), horizontal panning(yaw) and tilt (roll) of the virtual camera, as well as zoom.

Still another feature of the VNS is surgical instrument pathcorrelation. In particular, the VNS can compute the path taken by anendoscope scope (or other medical instrument) during a procedure andvarious related attributes and parameters, and can compute and display acorrelation between two paths.

I. Overall System Architecture and Operation

FIG. 1 is a high level diagram of a system for performing endoluminalsurgery. A flexible endoscope (“scope”) 1 is inserted into the body of apatient 2 through a natural orifice, such as the mouth 3. The scope 1includes a rigid base 4 and a flexible portion 5 which is inserted intothe body. The distal tip 6 of the scope 1 is part of the flexibleportion 5 and can be flexed about two or more orthogonal axes by thesurgeon, by using controls (not shown) mounted on the base 4. Thesurgeon navigates the scope 1 through a natural body lumen, such as theesophagus 7 and stomach 8 until the distal tip 6 of the scope 1 is inproximity to the target anatomy.

Optically coupled to the base 4 of the scope 1 is an endoscopic videocamera 9, which outputs a video signal to a display device (monitor) 10,which may be, for example, a cathode ray tube (CRT) display, liquidcrystal display (LCD), or other suitable type of display device.High-intensity light from a light source 11 is provided through a lightconduit to a light port on the base 4 of the scope 1 and is transmittedthrough the flexible portion 5 and output through the distal tip 6. Thescope 1 may include an instrument channel (not shown), through which asurgical instrument (such as a grabbing instrument for biopsies) can bepassed through to an opening at the distal tip 6. The entry port for theinstrument channel is normally on or near the base 4 of the scope 1.

As noted above, the VNS introduced here (not shown in FIG. 1) provides adisplay that includes coregistered views of intra-operative scan imagesand/or intra-operative scan images with live endoscopic video, amongother features. FIG. 2 illustrates schematically how such a display maybe presented to a user. The VNS may include its own display device, onwhich such a display can be presented. Alternatively, the display can bepresented on a separate external display device that is connected to theVNS 30, such as the monitor 10.

As shown in FIG. 2, a display 20 generated by the VNS includes severalwindows in proximity to each other, including a window 21 that containsa scan image and a window 22 that contains a corresponding live videoimage, presented side-by-side. The scan image is generated and updatedin real-time based on intra-operative scan data acquired during theendoscopic procedure from a CT scanning device, MRI device, ultrasonicimaging device, or other medical scanning device. The scan image iscoregistered with the live video image, according to the techniquedescribed herein. Other data, such as text, graphics, touchscreencontrols, etc., can be included on a separate portion 23 of the display20.

FIG. 3 schematically illustrates an endoscopic surgery visualizationsystem that includes the VNS 30. The VNS 30 receives intra-operativescan data from a medical scanning system 31, which can be, for example,a CT system, MRI system, ultrasonic imaging system in, or the like. TheVNS 30 also receives live video from the endoscopic video camera 9 thatis coupled to the scope 1. The VNS 30 also receives inputs 32 fromvarious sensors on the endoscope 1, which are used to determine thecurrent position and orientation of the distal tip 6 of the endoscope 1.The VNS 30 may also input pre-operative scan data from a data storagefacility 33 (e.g., a computer hard drive, file server, or the like). Thepre-operative scan data can be coregistered with the intra-operativescan data and or the live video.

The VNS 30 may have speech recognition/voice response capability; inthat case, the VNS 30 further receives audio inputs from a microphone34, through which to receive voice commands. The VNS 30 may alsoreceives various other user inputs 35, such as from touchscreen controlsor other input devices such as a keyboard, mouse, buttons, switches,etc. The VNS 30 outputs coregistered images such as described above toits own display device, if it is so equipped, and/or to an externalmonitor 10. The VNS 30 may also output synthesized speech and/or otherforms of audible output (e.g., warnings or distance to target) to theuser through an audio speaker 36. The VNS 30 may also include a networkinterface 37 through which to transmit and/or receive data over anetwork, such as a local-area network (LAN), a wide area network (WAN),a corporate intranet, the Internet, are any combination thereof. The VNS30 may also include a separate video camera and appropriate software(not shown) to capture and recognize gestures of the user as commands,in real-time, and to cause corresponding actions to be performed.

FIG. 4 is a block diagram showing the major subsystems of the VNS 30,according to certain embodiments of the invention. As shown, the VNS 30includes a data acquisition subsystem 41, a scope tracking subsystem 42,a measurement subsystem 43, a data processing subsystem 44, aregistration subsystem 45 and a user interface subsystem 46. The purposeof the data acquisition subsystem 44 is to load intra-operative andpre-operative scan data representative of a region of interest of agiven patient. The purpose of the registration subsystem is to bring thevarious data acquired into a common coordinate space. The registrationsubsystem 45 determines the transformation parameters needed tocoregister the data acquired by the data acquisition subsystem 41 andthe tracking subsystem 42 to a common coordinate space. These parametersare passed to the data processing subsystem 44, which transforms theinput data, performs segmentation and creates the desiredvisualizations. The data processing subsystem is the main processingsubsystem of the VNS 30.

The visualizations are passed to the user interface subsystem 46 foraudio and visual output. The user interface subsystem 46 also interpretsand passes user commands received in the form of any one or more of:voice, gestures, touch screen inputs, button presses, etc.

The purpose of the endoscope tracking subsystem 42 is to capture, inreal-time, data indicative of the position and orientation of theendoscope, particularly its distal tip, to enable coregistration ofmulti-modal images. Note, however, that the techniques introduced herecan also be used to track a surgical instrument other than an endoscope,such as a catheter, guide wire, pointer probe, stent, seed, or implant.

The measurement subsystem 43 receives user inputs and processed data viathe data processing subsystem 44, computes measurements of anatomicalfeatures, and formats the results to be passed to the user interfacesubsystem 46 for audio and/or visual output. These subsystems aredescribed further below.

FIG. 5 illustrates an example of a process that can be performed by theVNS 30, according to certain embodiments of the invention, while the VNS30 is in an operating mode to coregister scan images and live videoimages. Initially, the VNS 30 concurrently inputs intra-operative scandata, position/orientation data from the sensors on the scope, and livevideo from the endoscopic video camera, at 501 a, 501 b and 501 c,respectively. At 502 the VNS 30 determines the current position andorientation of the scope's distal tip. At 503 the VNS 30 generatesreal-time 3-D (volumetric) scan images, based on the intra-operativescan data and the current position and orientation of the scope's distaltip. The VNS 30 then coregisters the real-time 3-D scan images with thelive video from the endoscopic video camera at 504. The coregisteredscan images and live video are then sent to a monitor (which may beintegral with or external to the VNS 30) for display, to an imagerecording device for recording, and/or to a network interface fortransmission over a network. The process then repeats from the beginningwith new data while the VNS 30 is in this operating mode. It should beunderstood that the process of FIG. 5 is illustrated and described hereat a conceptual level; consequently, the exact sequence of operationsshown in FIG. 5 does not necessarily have to be the actual sequence inpractice. For example, input data can be received (501 a, 501 b and 501c) and buffered as necessary while the other operations in the processare being performed on previously received data (i.e., pipelinedprocessing).

FIG. 6 shows the architecture of the VNS 30 in greater detail. Thesubsystems of the VNS 30 shown in FIG. 4 will now be further describedwith reference to FIG. 6.

The purpose of the data acquisition subsystem 44 is to load scan datarepresentative of a region of interest of a given patient. Thissubsystem includes the following three modules:

-   -   1) An interface 61 to an intra-operative imaging device (e.g.,        CT device, MRI device, positron emission tomography (PET)        device, fluoroscopy device, ultrasound device) to receive        real-time intra-operative scan data from such device and to        configure and read the scan data. This can be, for example,        software within the imaging device.    -   2) An image reader module 62 to read medical images stored in,        for example, DICOM format, for loading pre-operative scan data.        The pre-operative scan data can be from, for example, an MRI        scan, CT scan, PET scan, fluoroscopic scan, or ultrasound scan.    -   3) A video interface 63 to the endoscopic video camera feed, to        receive and capture real-time intra-operative video of the        patient's anatomy. This interface can be, for example, a        Firewire interface, Universal Serial Bus (USB) interface, RS-232        interface, or the like, along with a frame grabber and        appropriate software to package frames as a real-time feed.

The purpose of the endoscope tracking subsystem 42 is to capture, inreal-time, data indicative of the position and orientation of theendoscope, particularly its distal tip. Note, however, that thetechniques introduced here can also be used to track a surgicalinstrument other than an endoscope, such as a catheter, guide wire,pointer probe, stent, seed, or implant.

The data processing subsystem 44 is the main processing subsystem of theVNS 30. In the illustrated embodiment, this subsystem includes an imagereslicer 64, a graphical model generator 65, an affine transform module66, a rigid (similarity) transform module 70, a barrel-distortioncorrection module 67, a multi-dimensional video generation module 78,and a path correlation module 79.

The image reslicer 64 produces reformatted images from scan data todesired positions and orientations. A reformatted image is derived byarbitrarily orienting a plane in 3D space, and assigning values to each2D pixel of the slice by interpolating the 3D voxels of the volume dataintersected by the plane.

The graphical model generator 65 generates surface models of theanatomical region of interest from the patient scan data provided by thedata acquisition subsystem 41. This module provides two types of images:

-   -   1) a volumetric perspective image which is rendered from a point        of view that correlates to one of the position/orientation        sensors attached to the endoscope.    -   2) a volumetric perspective image which is rendered from a point        of view that correlates to the position and orientation of the        patient. This will allow the surgeon to have, in effect, “x-ray        vision”, to see the neighboring anatomy relative to the position        of the surgical instrument.

The graphical model generator 65 provides segmentation functions such asthresholding, automatic detection of borders, creation of objects withinthe volume, extraction of surfaces, and further provides visualizationfunctions, e.g., rendering using different parameters (depth shading,gradient shading, maximum intensity projection, summed voxel projection,surface projection, transparency shading).

FIG. 9 is a block diagram showing the graphical model generator 65according to certain embodiments of the invention. The segmentationparameters, such as threshold, histogram, region of interest, shape aprioris, etc., are loaded from a file for that anatomical region. Theinput image volume 81 is cropped (cut) to the specified region ofinterest (ROI) by a Cut ROI unit 83. The segmentation unit 84 thenapplies a segmentation algorithm on the cropped volume. An example ofthe segmentation algorithm is the well-known seeded region growing.Other segmentation algorithms, such as level sets, can also be used. Theoutput of the segmentation unit 84 is a label map that specifies whichpixels belong to the segmented object and which do not. The label map issent as input to the surface generation unit 85, which in one embodimentapplies the marching cubes algorithm, i.e., by using polygons (e.g.,triangles) to represent the segmented surface. Since the number oftriangles can be very large, the decimation unit 86 applies a decimationalgorithm to reduce the triangles and produce a smoother surface. Thetriangles are reduced so that more can fit into the graphics card memory(not shown).

In parallel with the above-described operations, the data representativeof the co-registered location and orientation of the tracked scope ispassed as input to the model fitting unit 87. The model fitting unit 87fits a predetermined model to the data received. In one embodiment themodel used to represent the tracked surgical instrument is a line. Themodel fitting unit 87 produces model parameters 88, such that theposition and orientation of the line (i.e., the model) is the same asthat of the scanned data. The model parameters 88 are then sent to themerge module 74 for rendering.

Referring again to FIG. 6, the transform modules 66 and 70 apply ageometric transform to 3D points. The inputs expected by the transformmodules 66 and 70 are an image, a transform and the output of aninterpolation function. The interpolation is required since the mappingfrom one space to another will often require evaluation of the intensityof the image at non-grid positions. Nearest neighbor interpolation is anexample of the type of interpolation that can be used in this regard.

The transform modules 66 and 70 include affine transform module 66 andsimilarity transform module 70. A similarity (or “rigid”) transform isdefined as a transformation that preserve magnitudes of all lengths andangles. An affine transform is defined as a transformation whichpreserves parallelism of lines and includes rotation, scaling, shearingand translation. Each of the transforms is specified by an N×N matrixand an N×1 vector, where N is the space dimension. The number ofparameters is (N+1)×N. The first N×N parameters define the matrix incolumn-major order (where the column index varies the fastest). The lastN parameters define the translation for each dimension. The number ofdimensions is three (3).

The barrel distortion correction module 67 corrects the barreldistortion inherent in endoscopic video in order to facilitate accuratemeasurements and one-to-one comparison with visualization. Barreldistortion is a divergence from the rectilinear projection in geometricoptics where image magnification decreases with increasing distance fromthe optical axis. This type of distortion is a lens aberration or defectthat causes straight lines to bow outward, away from the center of theimage. The inverse mapping function used to correct the barrel lensdistortion can be determined a priori, or it can be obtained from themanufacturer of the endoscopic video camera.

The multi-dimensional video generation module 78 processes video framesacquired by the endoscopic video camera, to enable a user to navigatethe captured video with six degrees of freedom. This feature isdescribed in detail below.

The path correlation module 79 computes the path taken by the scope (orother medical instrument) during a procedure and various relatedattributes and parameters, and can compute and display a correlationbetween two paths. This feature is also described in detail below.

The purpose of the registration subsystem 45 is to bring the variousdata acquired into a common coordinate space. This subsystem includes amulti-modal image coregistration module 68 and a surgical instrumentregistration module 69.

The multi-modal image coregistration module 68 coregisters pre-operativepatient scan data with intra-operative patient scan data. Imagecoregistration is the process of determining the spatial transform thatmaps points from one image to the homologous points on a second image.FIG. 7 is a block diagram showing an example of an implementation of themulti-modal image coregistration module 68. Input data to thecoregistration process includes two images: one is defined as the “fixedimage” f(X), which may be, for example, an MRI image; the other isdefined as the “moving image” m(X), which may be, for example, a CTimage. The output of the multi-modal image coregistration module isparameters of an affine transformation matrix. The fixed image f(X) isthe intra-operative patient scan data and the moving image m(X) is thepre-operative patient scan data. The transform module T(X) representsthe spatial mapping of points from the fixed image space to points inthe moving image space. The metric module 70 performs a function S(f, m°T) to provide a measure of how well the fixed image is matched by thetransformed moving image and outputs this as a fitness value to theoptimizer 71. The optimizer 71 continues invoking the affine transformmodule 66 with different parameters until an optimal value for themetric has been reached or a maximum allowed number of iterations iscomplete. In one embodiment, the well-known Mutual Information basedImage-to-Image Metric is used for the metric module; affinetransformation is used for the transformation module; and gradientdescent optimization is used for the optimizer. Low-pass filtering ofthe images can be used to increase robustness against noise. In suchcases the low-pass filter can be a Gaussian image filter.

When an affine transformation is applied, many pixels in the outputimage do not have a corresponding input. That is, the correspondinginput falls in the middle of other voxels. The B-spline interpolator 72is therefore used to interpolate the voxel value at the output.

Referring again to FIG. 6, the surgical instrument registration module69 brings the data acquired by the tracking sub-system 42 into a commonco-ordinate space. This module takes data indicative of the position andorientation of the endoscope when placed on the origin of theco-ordinate system of the intra-operative imaging device as theparameters of the rigid transformation matrix.

The user interface subsystem 46 takes user input from, and providesaudio and visual output to, the user (e.g., the surgeon). This subsystemincludes a user input module 73 and a merge module 74. The merge module74 mixes graphical models from the graphical model generator 65, imageslices from the image reslicer 64, endoscopic video from endoscopicvideo camera (via the barrel lens distortion correction module 67), andtext at the desired position, orientation, resolution, and opacity, andproduces an image. The displays generated by the merge module 74 includethe following types of windows, any two or more of which can bedisplayed simultaneously and coregistered with each other:

-   -   1) Three orthogonal slices of the volume based on scan data:        transversal slice, sagittal slice, and coronal slice.    -   2) A rendering window that includes a volumetric perspective        image, based on scan data, rendered from a point of view that        correlates to the outputs of the position/orientation sensors        attached to the endoscope (i.e., the endoscopic camera point of        view).    -   3) A rendering window that includes a volumetric perspective        image, based on scan data, rendered from a point of view that        correlates to the position and orientation of the patient (not        necessarily the camera point of view; e.g., from the surgeon's        eye point of view).    -   4) A video image from the endoscopic video camera (after        barrel-distortion correction).

FIG. 8 is a block diagram showing the user input module 73 of the userinterface subsystem 46. The user input module 73 includes aspeech-to-text converter 75, a gesture recognition module 76, and aninput module 77 to receive the outputs of those modules and to receiveuser inputs from camera controls on the endoscopic video camera and fromcontrols on the VNS 30.

In operation, data generated by the intra-operative scanning device iscoregistered with the pre-operative scan at the beginning of theendoscopy procedure. The output of the coregistration is affinetransform parameters. This transformation is applied to thepre-operative scan data on iterations of intra-operative dataacquisitions. The coregistered patient scan is processed forvisualization, i.e., model generation of anatomical regions of interest.The model is sent to the merge module 74 for mixing with other models,images and text. The position/orientation data from the sensors on thescope is also registered to the common co-ordinate space and sent to themodel generator to provide a simulation of the surgical instrument, tobe mixed by the merge module 74.

Data from the intra-operative scan is also processed for visualization,i.e., model generation of anatomical regions of interest. The model issent to the merge module 74 for mixing with other models, images andtext. The image reslicer module 64 generates slices of the pre-operativeor intra-operative scan. The selection of the current slices(transversal, coronal, sagittal) is done either by user-input orautomatically by the tracking subsystem using a defined position on theendoscope, e.g., the distal tip, as reference. The video from theendoscopic camera feed is corrected for barrel lens distortion and thensent to the merge module 74 for mixing with other models, images andtext.

As noted above, the tracking subsystem 42 can automatically detectmovements of the flexible endoscope during an endoscopic procedure.Specific techniques and apparatus for detecting the current position andorientation of the scope are described below. In response to scopemovements, the data processing subsystem 44 can also automaticallyidentify a particular slice of intra-operative or pre-operative scandata corresponding to the current location and orientation of theendoscope tip and cause an image of that slice to be displayed to theuser from the viewpoint of the scope tip or another viewpoint, and inthe same way cause other slices of intra-operative or pre-operative scandata to be identified and displayed automatically in response to furthermovements of the endoscope during the procedure.

An example of a display that can be generated using this technique isshown in FIG. 22, in which three orthogonal slice images (e.g.,transverse, saggital and coronal planes) from intra-operative orpre-operative scan data are displayed coregistered with a 3D volumetricimage of a portion of a colon. Also displayed is a computer-generatedrepresentation of the distal end of the scope, so that the user can seethe relationship between the current viewpoint and the endoscopic cameraviewpoint. Although the displayed images are selected based on thecurrent position and orientation of the scope tip, the viewpoint of thedisplayed images can be a viewpoint other than the endoscopic cameraviewpoint, as shown in FIG. 22.

Multi-Planar Reconstruction (MPR) can be used to facilitate thisprocess. MPR is a well-known post-processing technique that canreconstruct axial scan images into coronal, sagittal and obliqueanatomical planes. The same technique can also be used to obtain slicesat any position and orientation through the volume. The data processingsubsystem 44 can use the MPR algorithm to obtain the slices at theposition and orientation of the tip of the scope and display them inreal-time. The end result is that when the user moves the scope, theslices corresponding to that position and orientation are automaticallydisplayed both over the 3D model and in the coronal, axial and sagittalwindows.

II. Scope Tracking

As noted above, the VNS 30 uses signals from position/orientationsensors on the scope to track the current position and orientation ofthe distal tip of the scope. Techniques to accomplish this will now bedescribed in greater detail.

Referring that FIG. 10, in accordance with a first embodiment, anelectromagnetic field generator is used as the center of the trackedvolume and is hence placed under the patient table 101 at a configurablelocation. The location of the generator constitutes the origin 102 ofthe tracking system. In certain embodiments, each sensor 104 on thescope 1 contains three small wire coils (not shown) orientedperpendicular to each other. The electromagnetic field 103 generated bythe generator induces current through these coils. The currents in thesecoils at any particular instant in time is dependent upon theelectromagnetic field strength at the location of the sensor and theorientation of the sensor relative to the generator. Hence, the distanceand orientation of the sensor 104 relative to the generator can bedetermined from those currents. Electromagnetic sensors such as theMicroBird sensors from Ascension Technology Corporation of Burlington,Vt., for example, are believed to be suitable for this purpose.

The currents produced by the coils in each sensor 104 are thentransmitted by one or more thin wires 105 to a pre-amplifier 106 in thescope, which amplifies the current. The pre-amplified current is sent toa signal processor (not shown) within (or used by) the trackingsubsystem 42 of the VNS 30, which computes the position and orientationof the sensor relative to the generator. The signal processor can be,for example, a conventional programmable microprocessor, digital signalprocessor, microcontroller, or other suitable processing device.

In certain embodiments of the invention, instead of using a separatechannel for the sensor output wires 105, the light channel 108 of thescope 1, which is used to transmit light from the light source 11 to thescope tip, is used is used as a conduit for the sensor output wires 105,as depicted in FIG. 10. Since the dimension of the sensor output wires105 is much smaller than the diameter of the light channel 108,embedding the wires 105 in the light channel 108 has negligible effecton the light transmitted and has the advantage of not requiring anyprotective coating over the transmitting wire 105, since the walls ofthe light channel serve that dual purpose.

Note that electromagnetic tracking of the scope may be susceptible tointerference when operating in the vicinity of CRTs, MRI scanningdevices or other devices that produce magnetic fields, as well as metalobjects such as office furniture, that disrupt magnetic fields. Also,with electromagnetic tracking devices the working volume tends to berelatively small. Furthermore, electromagnetic tracking is expensive andsensitive to errors because of the complex signal processing involved.

Consequently, in alternative embodiments of the invention, optics areused to track the scope's distal tip rather than electromagneticsensors. Optical tracking is advantageous, because it is benign, freefrom electromagnetic interferences, robust, and inexpensive. Opticaltrackers in general have high update rates, and sufficiently shortdelays. However, they are limited by line-of-sight problems, in that anyobstacle between the sensor and the source can seriously degrade thetracker system's performance. To overcome this problem, a hybrid opticaltracking approach can be used.

In the hybrid approach, the base 4 of the flexible scope 1 can betracked using conventional line-of-sight (e.g., LED based) opticaltrackers. In one embodiment, two cameras 110 are mounted on the ceilingor on a fixed frame 111, and several light emitting diodes (LEDs) 112are placed at fixed, known positions on the base 4 of the scope, asshown in FIG. 11. Projections of the light emitted by the LEDs 112 ontothe camera image planes 113 contain enough information to uniquelyidentify the position and orientation of the base 4 of the scope, usingwell-known methods. Various photogrammetric methods, particularlytriangulation, can be used to compute this transformation and obtain the3D coordinates of the base of the scope. These coordinates serve as areference point for determining the position and orientation of thedistal tip 6 of the scope 1.

For purposes of scope tracking, the LEDs 112 can be referred to in moregeneral terms as location elements. As an alternative to LEDs 112, othertypes of location elements could be used, such as essentially any othertype of device or devices by which the current position of the base 4can be determined. Other types of location elements might include othertypes of light-emitting devices, one or more radio frequency (RF)transmitters, or even passive location markers such as reflective tags.In such alternative embodiments, the cameras 110 may be replaced by adifferent type of device, to be compatible with the type of locationelement(s) used, and may be an active device such as an RF transmitterif the location elements are passive. As another example, video camerasin combination with sophisticated shape recognition software can be usedto identify and precisely determine the current position and orientationof the base 4 of the endoscope, without using any location elements onthe base 4.

The flexible portion 5 of the scope 1 can be tracked by using opticalfiber curvature sensors. As represented schematically in FIG. 12, anoptical fiber curvature sensor 120 includes a light source 121, opticalfibers 122 to carry light from the light source 121, a connector 125 forthe light source 121, a photodetector 126 to measure the intensity oflight transmitting through the optical fibers 122, a connector 127 forthe photodetector 126, an amplifier (not shown) to amply the sensedsignal and a signal processor (not shown). A typical phototransistor canbe used as the photodetector 126. It is desirable that the selected typeof photodetector be the type most sensitive to the wavelength of thelight emitted from the light source 121. The output signal 128 from theoptical curvature sensor can be transmitted back to the trackingsubsystem of the VNS 30 via thin wires rounded through the lightchannel, as described above in relation to FIG. 10.

Note that in certain embodiments of the invention, some of theabove-mentioned elements may be external to the optical curvature sensor120; for example, the output signal amplifier and/or the signalprocessor can be in the tracking subsystem 42 of the VNS 30. As anotherexample, the external light source 11 for the endoscope 1 can also serveas the light source 121 of all of the optical curvature sensors 120 inthe endoscope 1. In that case, the light source connector 125 on eachsensor 120 is connected with the light transmitting channel 108 (FIG.10) of the endoscope 1, as shown in FIG. 13, to optically couple eachsensor 120 to the light source 11.

Since the output signal of the sensor corresponds to the average bend ortwist in the sensor, depending on the type of the curvature sensor used,the length of the sensor is an important consideration. Hence, using asingle sensor as long as the flexible portion 5 of the scope is notadvisable, because the end-tip accuracy will be low. Instead, multiplefiber optic curvature sensors 120 can be placed in along the flexibleportion 5 of the scope 1. As shown in FIG. 14, optical curvature sensors120 of variable lengths can be placed along the flexible portion 5 ofthe scope 1, instead of using sensors of the same length. It is known apriori that the flexible portion 5 of the scope tends to deflect most atthe distal tip 6 and least where it is closer to the base 5. Withincreasing proximity towards the distal tip 6, the possible deflectionincreases, and hence, the length of the curvature sensors 120 shoulddecrease, as shown in FIG. 14.

Based on the data of the state of curvature at each sensor 120 and theknown separation between sensors 120, the signal processing device inthe VNS 30 can determine the shape, position and orientation of theflexible portion 5 of the scope 1. Those computed coordinates are withrespect to a reference point, which in this case is the coordinates ofthe base 4, computed as described above. Therefore, if sufficientcurvature measurements are taken and appropriately integrated, the exactposition and orientation of the distal tip 6 of the scope can becomputed, relative to the origin 102 of the tracking system (FIG. 10).This information can then be used to coregister live video from theendoscopic camera 9 with intra-operative and/or pre-operative scanimages as described above.

FIG. 15 illustrates an example of the process that may be performed bythe tracking subsystem to determine the current position and orientationof the distal tip 6 of the scope 1, according to the hybrid approachdiscussed above. At 1501A, the VNS 30 receives signals from theposition/orientation sensors or LEDs 112 on the base 4 of the scope 1.Concurrently with 1501A, at 1501B the tracking subsystem 42 receivessignals from the position/orientation sensors 120 on the flexibleportion 5 of the scope 1. At 1502A the tracking subsystem 42 thencomputes the current position of the base 4 of the scope 1 as areference point (relative to the origin 102 of the tracking system),based on the signals from the sensors/LEDs 112 on the base, andconcurrently at 1502B, it computes the current position and orientationof the distal tip 6 of the scope 1, relative to the reference point. At1503, the tracking subsystem 42 computes the current position andorientation of the distal tip 6 of the scope 1 relative to the origin102 of the tracking system, based on the current computed referencepoint and the current computed position and orientation of the distaltip 6 of the scope 1 relative to the reference point. The process thenrepeats using new inputs from the sensors.

III. Multi-Dimensional Navigation of Video

During endoscopic surgery, the video that is acquired from theendoscopic video camera 9 is a sequence of images captured through thescope 1, while the scope 1 is pointed in different directions atdifferent instances in time. The video can be recorded as it isacquired. When playing back that video in the prior art, there is noknown way to navigate through that video except to play it frame byframe, which provides only a single degree of freedom for visualnavigation, i.e., time. Time is not always the best dimension in whichto view the video, since it forces video playback from the scope's pointof view.

It is therefore desirable to provide multi-dimensional navigation, i.e.,visual navigation of endoscopic video with multiple degrees of freedom,or dimensions, i.e., not just time. The other navigation dimensions thatthe technique introduced here adds are position and orientation. Moreprecisely, the technique which will now be introduced provides sixdegree of freedom for visual navigation of endoscopic video. In effect,this technique allows the user control of a virtual camera (point ofview) that can be translated in three orthogonal axes in 3-D space aswell as allowing control of vertical panning (pitch), horizontal panning(yaw) and tilt (roll) of the virtual camera, as well as zoom.

This technique will be described now is reference to FIGS. 16 through19. Initially, the position x(t) and orientation v(t) of each capturedvideo frame 160-i or image is determined when the frame is captured, andthe frame is tagged with this information, as shown in FIG. 16. Hence,the acquired video becomes a set of images with known positions andorientations.

Referring to FIG. 17, given the position X and unit orientation vector Vof the scope's distal tip 6, the position X′ and unit orientation vectorV′ of the image plane 171 can be computed as X′=X+f.V′ where X=[x,y,z]T,and V=[i,j,k] T and f is the distance between the distal tip 6 and theimage plane 171.

V′=V since the normal to the image plane 171 is the orientation vectorof the distal tip 6.

X′ and V′ can then be used to place the image plane 171 in virtual 3Dspace at the position X′ and the orientation V′. The OpenGL and VTKsoftware libraries can be used to place the image at the specifiedposition and orientation in space.

Hence, as the scope tip's position and orientation information isobtained, a frame or image 160-i is grabbed from the video feed andplaced in the 3D window coordinate system 180, as illustrated in FIG.18. Also text indicating the timestamp can be overlaid on each frame160-i, so that the user will know at what instant the frame was grabbed.

Thus, at any given point in the surgery, when the user wishes tonavigate through the grabbed frames, he navigates through the 3D spaceby using a mouse, keyboard, or other user input device. The classvtkRenderWindowInteractor in the VTK library captures mouse and keyboardevents in a render window. Accordingly, mouse and keyboard events can beapplied to control the viewpoint and the orientation of the virtualcamera associated with the render window in which the frames or imagesare being placed. This in effect allow the user to navigate through thevideo with six degrees of freedom, i.e., translation along all threecoordinate axes as well as pitch, yaw and roll, of the virtual camera.In order to provide smooth transitions between intersecting imageplanes, interpolation can also be employed. For example, bilinearinterpolation, nearest neighbors interpolation or other types ofinterpolation can be used.

FIGS. 19A and19B show processes associated with this navigationtechnique. These processes are performed by the multi-dimensional videogeneration module 78 (FIG. 6). More specifically, FIG. 19A shows aprocess of acquiring and processing video data in order to enablesubsequent visual navigation as described above. At 1901 the VNS 30acquires video frames from the endoscopic camera video feed. At 1902 theVNS 30 determines the position and orientation of the distal tip of thescope for each frame as it is acquired. At 1903 the VNS 30 associatesthe corresponding position and orientation information with each framein memory. An at 1904 the VNS 30 computes, in the manner describedabove, the position and orientation of the image plane for each of thevideo frames, as a function of the corresponding position andorientation of the distal tip for each frame.

FIG. 19B shows the process by which the user can visually navigatethrough video that has been processed as described above. Using thisprocess, a user can navigate previously-recorded processed video, or auser can navigate video as it is being acquired, in near real-time(subject only to the delay required to process the video as described inreference to FIG. 19A). At 1921 the VNS 30 receives user inputspecifying a visual navigation action. The action may be, for example, amouse input or joystick input specifying movement of the virtual camerain translation and/or in rotation relative to any one or more of thethree coordinate axes. In 1922 the VNS 30 identifies the frame or framesthat are affected by the user input. The VNS 30 then transforms theimage plane for each affected frame to a new spatial position andorientation, based on the user input, as described above. The VNS 30then causes each affected frame to be displayed according to its newimage plane at 1924.

IV. Model Fitting Measurement Technique

It is desirable to be able to obtain in vivo measurements of anatomicalfeatures in the body during an endoscopic procedure. For example, duringendoscopic surgery it may be desirable to know the size of a colonpolyp, such as polyp 201 shown in FIG. 20. If an incision needs to bemade, it is desirable to know the required size of the incision so thatthe instrument used for making the incision can be selected orconfigured accordingly. Alternatively, if there is a need to ablatetissue, then it is desirable to know the size of the region that needsto be ablated, so that the ablation instrument strength and range can beconfigured. If an implant needs to be placed inside the body, such as inthe esophagus for the GERD procedure or a heart valve implant needs tobe positioned, it is desirable to know the size of the opening so thatthe implant's size, orientation and position can be selected.

With this in mind, the VNS 30 can also include capability to obtainapproximate measurements of an anatomical feature or region of interestin vivo, through the measurement subsystem 43. The approach introducedhere includes fitting a user selected implicit model to a set of pointsthat reside on the surface of the anatomical region of interest.Implicit functions are of the form F(x,y,z)=constant.

An example of an implicit function that can be used is a sphere S ofradius R and centered at the origin, which can be described by theequation F(x,y,z)=R2−x2−y2−z2. The equation F(x, y z)<0 describes asphere that lies inside the sphere S, and when F(x,y,z)>0, a sphere thatlies outside the sphere S is defined. The unknown in the above implicitfunction is the radius R of the sphere. Note that the techniquedescribed here is not restricted to one implicit function; a list ofimplicit functions may be made available, from which the user canchoose.

The set of points needed to fit the implicit function are initiallycollected by the user's pressing a predetermined control input (e.g.,pressing a designated button on the scope 1 or the endoscopic videocamera 9) when the tracked distal tip 6 of the scope is touched to threeor more locations of the surface of the anatomical feature of interest,such as locations 202, 203 and 204 on the surface of the polyp 201 shownin FIG. 20. FIG. 20 illustrates an endoscopic camera view of a polyp.

The implicit model fitting problem can be phrased as follows:

The goal is to approximate a real valued function f(x) by s(x) given theset ofX={χ ₁, . . . , χ_(N) }⊂R ^(d)values f=(f1, . . . , fN) at the distinct points

For the fitting process a least squares schema can be used, thatminimizes the distance between the global implicit surface fieldfunction and the 3-D points. Orthogonal distance regression can be usedto minimize the distance function, where the sum of squares oforthogonal distances from the data points to the surface is minimized.An alternative approach can use radial basis functions to approximatethe implicit surface. Note that a product FastRBF from ARANZ can be usedto smoothly interpolate scattered 2D and 3D data with Radial BasisFunctions (RBFs).

Once the model is fitted, it is meshed and displayed to the user alongwith the associated parameters. The user can mark additional points onthe surface to improve the accuracy of the model; in that sense, themodel and this approach in general are adaptive.

The user can also perform binary operations of union and intersectionbetween models. One possible use of such an operation is to allow thesurgeon to see how the model of an anatomical feature intersects with agraphical model of a surgical implant. Measurements from the anatomicalmodel will be the attributes associated with the selected implicitfunction, such as volume, surface area, minimum extent, maximum extent,etc.

FIG. 21 illustrates an example of this process. At 2101 the VNS 30receives user input specifying three or more points on the surface of ananatomical feature. As noted above, this can be done by the user'stouching the distal tip 6 of the scope 1 to the surface of theanatomical feature and providing a predetermined control input at eachpoint to command the system to capture the coordinates of the point. At2102 the tracking subsystem 42 computes the 3D locations of the capturedpoints. The measurement subsystem 43 then fits a model volume (e.g., asphere), as defined by the user-selected implicit function, to thecaptured points. At 2104 the measurement subsystem computes a mesh toform the surface of the model volume. At 2105 the VNS 30 causes thesurface to be displayed to the user, and the measurement subsystem 43computes one or more physical parameters of the anatomical feature asrepresented by the model surface, such as its volume. The parametersthat can be computed depend on the implicit model that is used. If theimplicit model is a sphere, then another parameter that can be computedto reflect an attribute of the anatomical feature is the surface area,for which a standard formula can be used. Other parameters can also becalculated using standard formulas, depending on the implicit model.

The computed parameter or parameters are then output to the user at 2106by the user interface subsystem 46. The user can at this point specifyone or more additional points on the surface of the anatomical featureto refine the model and, hence, the measurements. If no such additionaluser inputs are received (2107), the process ends.

If additional user inputs specifying points are received, the model isthen refit to all of the specified points at 2108, and the surface ofthe refitted model is then computed at 2109. The new surface isdisplayed to the user and the parameter or parameters are recomputed at2110. The recomputed parameter or parameters are then output to the userat 2111.

V. Path Computations and Display

As noted above, the data processing subsystem 44 in certain embodimentsincludes a path correlation module 79 (FIG. 6) that can compute the pathtaken by the scope (or any other surgical instrument) during a procedureand various attributes and parameters related to that path. The pathcorrelation module 79 receives data indicating the current position andorientation of the scope from the tracking subsystem 42. It converts thedata into a dataset representing the path taken by the scope, andtransforms that dataset into graphic model parameters to enable the pathto be rendered and displayed to a user in 3D. An example of how such apath can be displayed to a user by the VNS 30 is shown in FIG. 23. Thepath, such as path 230 in FIG. 23, may be shown in 3D and in color,where different colors are used to indicate the distance betweenparticular points on the path 230 and the viewing plane.

The path correlation module 79 can also determine a correlation betweentwo or more paths, e.g., between the actual path of the scope (or othermedical instrument) and a predefined reference path, and can display thetwo paths and output an indication of that correlation to a user. FIG.24 shows an example of how the spatial relationship between two pathscan be displayed. More specifically, FIG. 24 shows an example of adisplay that includes a 3D rendering of a colon 240, a pre-surgeryplanned path 241 for the endoscope, and the actual path 242 taken by theendoscope (which may be determined in real-time or based on previouslyrecorded movements of the scope). With such a display, the user caneasily see the deviation of the actual path 242 from the planned path241.

The correlation may be determined as the amount and location ofdeviation between a given position and orientation of the scope from thereference path. More specifically, this may be determined by firstfinding a position along the reference path which is closest to acurrent position of the scope, and then determining the deviation as thedistance between the current location of the scope and that positionalong the reference path. FIG. 27 illustrates the computation ofcorrelation between the current position of the scope tip 6 and areference path 271, where the correlation is represented by line 272.

The path correlation module 79 can also determine a correlation betweentwo recorded paths (e.g., an amount and location of deviation betweenthe two paths) and output an indication of that correlation to a user.An example of a process for determining a correlation between tworecorded paths is illustrated in FIG. 28.

Referring to FIG. 28, at 2801 the process projects both a first recordedpath and a second recorded path onto one of the three orthogonal imageplanes, i.e., x-y, y-z or z-x plane (this operation will also be donefor the other two planes, as described below). FIG. 25 illustrates anexample of projecting a path onto three orthogonal planes. Projection252 is the projection of path 251 onto the x-y plane, projection 253 isthe projection of path 251 onto the y-z plane, and projection 254 is theprojection of path 251 onto the z-x plane.

At 2802 the process computes a first line that connects the startingpoint of the first recorded path with the starting point of the secondrecorded path and, at 2803, computes a second line that connects the endpoint of the first recorded path with the end point of the secondrecorded path. Operations 2802 and 2803 are illustrated in FIG. 26, inwhich lines 261 and 262 connect the starting point and endpoints,respectively, of two recorded paths 263 and 264 projected onto a plane.

At 2804 the process computes an area, medial axis, of a shape, theboundary of which is collectively defined by the first and secondrecorded paths and the first and second lines. Operations 2801 through2804 are then repeated for the y-z and the z-x planes (2805/2807). At2806 the process computes the total area for all three iterations of2801 through 2804 as a measure of the correlation between the first andsecond recorded paths.

The path correlation module 79 can also identify a landmark along thepath (e.g., from previously recorded user input, such as a voice inputor a button click on the endoscopic video camera, identifying thelandmark) and metadata associated with the landmark (e.g., voice and/ortext), and transform an associated position and orientation of theendoscope and the metadata into graphics primitives, to allow renderingand display of data indicative of the associated position, orientationand the metadata. In addition, the path correlation module 79 can alsocompute the distance between the current position of the endoscope andthe landmark and cause an indication of the distance to be output to auser.

Thus, a visual navigation system for use in endoscopic surgery,particularly endoluminal surgery, has been described. The techniquesintroduced above, including all of the modules of the VNS 30, can beimplemented in logic such as special-purpose hardwired circuitry,software and/or firmware that runs on one or more programmableprocessors, or in a combination thereof. Special-purpose hardwiredcircuitry may be in the form of, for example, one or moreapplication-specific integrated circuits (ASICs), programmable logicdevices (PLDs), field-programmable gate arrays (FPGAs), etc.

Software or firmware to implement the techniques introduced above may bestored on a machine-readable medium and may be executed by one or moregeneral-purpose or special-purpose programmable microprocessors. A“machine-accessible medium”, as the term is used herein, includes anymechanism that provides (i.e., stores and/or transmits) information in aform accessible by a machine (e.g., a computer, network device, personaldigital assistant (PDA), manufacturing tool, any device with a set ofone or more processors, etc.). For example, a machine-accessible mediumincludes recordable/non-recordable media (e.g., read-only memory (ROM);random access memory (RAM); magnetic disk storage media; optical storagemedia; flash memory devices; etc.), etc.

Although the present invention has been described with reference tospecific exemplary embodiments, it will be recognized that the inventionis not limited to the embodiments described, but can be practiced withmodification and alteration within the spirit and scope of the appendedclaims. Accordingly, the specification and drawings are to be regardedin an illustrative sense rather than a restrictive sense.

1. A computer-implemented method comprising: tracking a current position of a surgical instrument through a body lumen during an endoscopic procedure; generating data indicative of the current position of the surgical instrument for a plurality of positions of the surgical instrument during the endoscopic procedure; storing said data as a dataset representing a first path taken by the surgical instrument through the body lumen during the endoscopic procedure; causing an indication of the first path to be displayed to a user based on the dataset; determining a correlation between the first path and a second path through the body lumen, wherein determining the correlation includes determining a deviation of a position of the surgical instrument from the second path by projecting the first path and the second path onto each of three orthogonal planes, computing for each of the projections a first line connecting a first point of the first path and a second point of the second path and a second line connecting a third point on the first path and a fourth point on the second path, and computing an area of a shape defined by each of the projections of the first path and the second path and the corresponding first and second lines; and outputting a summed total of the calculated areas as an indication of the correlation to a user.
 2. A method as recited in claim 1, further comprising: transforming the data indicative of the positions of the surgical instrument along the first path into graphic model parameters to enable the first path to be rendered and displayed to a user.
 3. A method as recited in claim 1, wherein the second path is a pre-surgery planned reference path.
 4. A method as recited in claim 1 wherein the second path is a second recorded path of the surgical instrument.
 5. A method as recited in claim 1, further comprising: during the endoscopic procedure, identifying a landmark along the first path and metadata associated with the landmark; and transforming an associated position of the surgical instrument and said metadata into graphics primitives to allow rendering and display of data indicative of the associated position, orientation and the metadata.
 6. A method as recited in claim 5, further comprising: during the endoscopic procedure, computing a distance between the current position of the surgical instrument and a landmark along the first path; and causing an indication of the distance to be output to a user.
 7. A method as recited in claim 1, wherein the surgical instrument is a flexible endoscope.
 8. A method as recited in claim 7, wherein tracking the current position of the surgical instrument during an endoscopic procedure comprises tracking the current position of a distal tip of the flexible endoscope.
 9. A system comprising: a processor; and a memory, coupled to the processor, storing instructions, which when executed by the system, cause the system to track a current position of an endoscope through a body lumen during an endoscopic procedure and to generate data indicative of the current position of the endoscope for a plurality of positions of the endoscope during the endoscopic procedure; determine a first path taken by the endoscope during the endoscopic procedure, based on said data indicative of the current position of the endoscope for the plurality of positions of the endoscope determine a correlation between the first path and a second path through the body lumen, wherein determining the correlation includes determining a deviation of a position of the surgical instrument from the second path by projecting the first path and the second path onto each of three orthogonal planes, computing for each of the projections a first line connecting a first point of the first path and a second point of the second path and a second line connecting a third point on the first path and a fourth point on the second path, and computing an area of a shape defined by each of the projections of the first path and the second path and the corresponding first and second lines; and cause a summed total of the calculated areas as an indication of the correlation to be displayed to a user.
 10. A system as recited in claim 9, the instructions further causing the system to: transform the data indicative of the positions of the endoscope instrument along the first path into graphic model parameters to enable the first path to be rendered and displayed to a user.
 11. A system as recited in claim 9, wherein the second path is a pre-surgery planned reference path.
 12. A system as recited in claim 9, wherein the second path is a second recorded path of the endoscope.
 13. A system as recited in claim 9, the instructions further causing the system to: identify, during the endoscopic procedure, a landmark along the first path and metadata associated with the landmark; and transform an associated position of the endoscope and said metadata into graphics primitives to allow rendering and display of data indicative of the associated position, orientation and the metadata.
 14. A system as recited in claim 9, the instructions further causing the system to: compute, during the endoscopic procedure, a distance between the current position of the endoscope and a landmark along the first path; and cause an indication of the distance to be output to a user.
 15. An apparatus comprising: means for tracking a current position of a surgical instrument through a body lumen during an endoscopic procedure; means for generating data indicative of the current position of the surgical instrument for a plurality of positions of the surgical instrument during the endoscopic procedure; means for storing said data as a dataset representing a first path taken by the surgical instrument through the body lumen during the endoscopic procedure; means for causing an indication of the first path to be displayed to a user based on the dataset; means for determining a correlation between the first path and a second path through the body lumen, wherein determining the correlation includes determining a deviation of a position of the surgical instrument from the second path by projecting the first path and the second path onto each of three orthogonal planes, computing for each of the projections a first line connecting a first point of the first path and a second point of the second path and a second line connecting a third point on the first path and a fourth point on the second path, and computing an area of a shape defined by each of the projections of the first path and the second path and the corresponding first and second lines; and means for outputting a summed total of the calculated areas as an indication of the correlation to a user.
 16. An apparatus as recited in claim 15, wherein the second path is a pre-surgery planned reference path.
 17. An apparatus as recited in claim 15, wherein the second path is a second recorded path of the surgical instrument. 